3.4. Memory

What does "memory" mean in this course?

You close the terminal and ask the same question tomorrow: which of these six stores still knows the answer?

The word covers six stores that should not be conflated:

Each row has a different scope, owner, and lifecycle, and confusing two of them is the classic memory bug. A note saved for one user is not knowledge the whole team can read, and an A2A task is not a conversation.

The last row is the one people reach for first and find last. "How do I keep a value across turns?" is answered by callback_context.state, a plain dict ADK persists with the session, where the key prefix decides the lifetime: no prefix persists with this session, temp: lives only for the current invocation and is never written back, user: spans every session of that user id, and app: is shared by every user of the app. budget.py is the shipped example — it stores budget:input_tokens and budget:output_tokens without a temp: prefix precisely so the running token totals survive the whole conversation rather than resetting each turn (7.3. Costs spends them). Choosing the prefix is choosing the store; there is no separate API to get wrong.

This page covers long-term memory and knowledge retrieval. A runbook — a committed Markdown procedure for one failure mode — is not automatically trusted simply because it is stored locally; its content is still data the model must cite and policy must constrain.

This page has two halves. The first covers the six stores and how a runbook is fetched, scored, and evaluated. The second is the one context window they all compete for, because storing memory and spending the window that holds it are two different problems.

What should an agent remember across sessions?

Facts the next conversation needs and cannot recompute: the incident under investigation, remediation already attempted, and outcome notes. In-session history dies with the session, and runbook retrieval only knows the knowledge base.

longterm.py gives the AgentOps Agent two tools — save_incident_note and recall_incident_context — backed by a small SQLite table in the disposable state directory. Notes are isolated per user and pass PII redaction before persisting:

# simplified
# Memory is a persistence boundary: redact before the write, not after the read.
redacted = redact_persisted_text(cleaned)

Memory is a data store like any other: it never touches the seed dataset, and mise run data:reset clears it along with the rest of the runtime state.

The cross-session boundary depends on a stable ToolContext.user_id. The default unauthenticated A2A adapter derives A2A_USER_<context-id>, so a new A2A context is a new logical user and cannot recall the previous context's notes. A browser reload in the minimal client is enough to cross that line.

A shared or production client must authenticate the caller and propagate a durable subject before treating this store as human-level long-term memory.

Why make memory explicit tools instead of automatic?

Silent context stuffing hides reads and writes from everyone debugging the agent. As tools, every recall and save appears in the trace, can be audited, and can be tested deterministically offline.

Explicit tools also keep the boundaries inspectable. Input validation runs at the tool edge, before anything is written:

• normalize_incident_id parses the id, so only a well-formed value reaches the store.
• A referential check refuses to attach a note to an unknown incident — "refusing to create orphaned memory".
• A length cap and an empty-note check reject the rest.

The redaction step above is then a visible policy, not a hope. The cost is that the model must decide to call recall_incident_context: both the system instruction and tool docstring tell it to do so at the start of an investigation, and a named trajectory gate verifies it does.

Why not ADK's MemoryService?

ADK ships a memory abstraction, so writing longterm.py instead was a decision, and it is worth being able to defend rather than inherit.

The shipped implementations sit at two extremes. InMemoryMemoryService loses everything when the process exits, which fails the one property this store exists for — surviving the session. VertexAiMemoryBankService is durable and genuinely good, but it is a hosted Google Cloud service, which fails the course's account-free, offline, required path before it fails anything technical.

Three requirements decided it, and each one is a property you can check:

1. Offline and account-free. A learner with no cloud project must be able to save a note and recall it tomorrow.
2. Auditable at the write. A note passes redact_persisted_text before it lands, and the tool refuses to attach a note to an unknown incident. A managed store that ingests session history automatically gives you no place to stand between "the model said it" and "it was persisted".
3. Explicit in the trace. Every recall and save is a tool call, so a trajectory evaluation can assert that recall happened. Automatic memory ingestion is invisible to exactly the check you most want.

Read that as a trade, not a verdict. A managed memory bank buys semantic recall across a large history that a SQLite table keyed by user id does not. If your deployment already lives on Vertex and you can meet requirement 2 at the ingestion boundary, MemoryService is the shorter path — swap the two tools for it and keep the eval case that proves recall still happens.

How is a known runbook fetched?

An incident carries a validated runbook slug. The tool normalizes it before any path lookup:

def get_runbook(slug: str) -> dict[str, Any]:
    """Fetch a runbook by its exact slug (e.g. an incident's ``runbook`` field).

    Args:
        slug: The exact runbook identifier returned by ``get_incident``, e.g.
            ``high-latency`` or ``service-down``. Never derive it from the service name.

    Returns:
        A dict with the ``slug`` and its markdown ``content``, or an ``error`` if unknown.
    """
    # Parse at the boundary: normalize once here, then work only with the trusted
    # value — read, error message, and result all speak the normalized slug.
    normalized = normalize_slug(slug)
    if normalized is None:
        known = ", ".join(data.list_runbook_slugs())
        return {"error": f"Invalid runbook slug {slug!r}. Available runbooks: {known}."}
    content = data.read_runbook(normalized)
    if content is None:
        known = ", ".join(data.list_runbook_slugs())
        return {"error": f"No runbook named {normalized!r}. Available runbooks: {known}."}
    return {"slug": normalized, "content": content}

The exact excerpt is build-checked against memory.py. normalize_slug permits only lowercase alphanumeric kebab-case, and only the trusted normalized value reaches the data layer. ../../secret never becomes a filesystem path.

get_runbook and search_runbooks ship together as KNOWLEDGE_TOOLS, each wrapped in with_resilience — the same bounded-deadline-and-retry policy the read tools get, safe here because a runbook read is idempotent.

Choose the narrowest lookup. If an incident record supplies runbook: high-latency, call get_runbook("high-latency"); use search_runbooks only when no trusted slug exists.

The same two functions are also two of the six read-only tools re-exposed over the governed MCP route (3.3. MCP), so one implementation serves the in-process agent and any external MCP client without a second copy to drift.

A runbook is plain Markdown split by ## headings, and high-latency has four: Symptoms, Diagnosis, Remediation, and Related. Here is its Remediation section — INC-001 and INC-005 both point at this slug, and this is the part the agent quotes when it recommends a fix:

## Remediation

- If a recent deploy correlates, **roll back** to the previous version (see `deployment-rollback`).
- Warm or repair the cache if hit-rate collapsed.
- Scale out replicas to shed load while you find the root cause.
- Raise connection-pool limits if the pool is the bottleneck.

# Runbook: High Latency

**Applies to:** a service whose request latency (p95/p99) has risen well above its normal baseline while still returning successful responses.

## Symptoms

- p95 or p99 latency several times higher than the 7-day baseline.
- Timeouts or slow responses reported by downstream callers.
- Error rate usually normal (requests succeed, just slowly).

## Diagnosis

1. Check whether the spike lines up with a recent **deploy** or config change.
2. Inspect CPU, memory, and thread/goroutine counts for saturation.
3. Look at **downstream dependencies** (database, cache, third-party APIs) for slow calls.
4. Check cache **hit-rate**; a cold or thrashing cache is a common cause after a rebuild.
5. Review connection-pool and queue depths for contention.

## Remediation

- If a recent deploy correlates, **roll back** to the previous version (see `deployment-rollback`).
- Warm or repair the cache if hit-rate collapsed.
- Scale out replicas to shed load while you find the root cause.
- Raise connection-pool limits if the pool is the bottleneck.

## Related

- `deployment-rollback` — reverting a bad release.
- `elevated-errors` — when latency is accompanied by failures.

The remediation skill's "follow that runbook's Remediation section" (3.2. Skills) refers to the section you see here — the runbook supplies the incident-specific what, the skill the reusable how.

How does free-text retrieval work?

search_runbooks scores every runbook against the words in your query, with no model involved. The scoring is deterministic and TF-IDF-style: a term counts for more when it is rare across the runbooks.

The scorer, in the order it runs:

• Drop tokens shorter than three characters.
• Count how many documents contain each term.
• Weight rare terms more highly than ubiquitous terms.
• Add a strong boost when a term matches the runbook slug.
• Sort by score descending, then slug ascending for stable ties.

# simplified
scored.sort(key=lambda row: (-row[0], row[1]))

The result is inspectable, fast, cheap, and reproducible. It is intentionally not described as semantic search — that is the opt-in branch below.

When do embeddings beat keyword retrieval?

When queries paraphrase the documents instead of quoting them — an engineer types "checkout is slow" while the runbook says "high latency". Whether that matters for this corpus is a measurement, not a fashion choice: enable semantic retrieval only if the evaluation below shows it beating the keyword baseline on this dataset.

An embedding is a vector representation of text, so a search can match meaning instead of exact words (0.7. Glossary).

The implementation in retrieval.py stays minimal and account-free. sqlite-vec, a SQLite extension for vector search, stores runbook-chunk vectors in the state directory, and the nomic-embed-text model runs through the local Ollama embeddings endpoint. Chunking — splitting each document into smaller passages — is heading-bounded, so each vector covers one procedure step or symptom block:

# simplified
def chunk_runbook(slug: str, content: str) -> list[str]:
    """Split one runbook into heading-bounded chunks, each prefixed with its slug."""
    pieces = [piece.strip() for piece in _HEADING.split(content) if piece.strip()]
    return [f"{slug}: {piece}" for piece in pieces] or [f"{slug}: {content.strip()}"]

The table's existence is not enough: old vectors can have the right schema and the wrong meaning. Before every semantic search, the implementation derives an index specification from the normalized corpus, configured embedding model, resolved model blob digest when Ollama exposes one, and chunker version. One metadata row binds the complete vector generation to eight reproducibility fields:

The first caller takes a SQLite BEGIN IMMEDIATE lock, builds the entire replacement generation inside that transaction, and commits only after every vector and metadata field validates. Concurrent callers wait and then reuse that generation. A failed rebuild rolls back, and the old complete generation remains stored but is not presented as current; the agent falls back to keyword retrieval instead of comparing incompatible vectors.

The option is off by default (AGENT_SEMANTIC_RETRIEVAL=false), which keeps the offline test gate deterministic and model-free. search_runbooks reads that one flag and branches: the keyword scorer above, or a lazily-imported semantic_search so the default path never loads the vector stack.

The fallback is explicit, not silent. An EmbeddingUnavailableError (endpoint down, model unpulled, mismatched batch) is caught, logged at warning, and answered by the same keyword scorer:

flowchart TB
    Q["search_runbooks(query, limit)"] --> C{"settings.semantic_retrieval?"}
    C -->|"false (default)"| K["keyword scorer<br/>rare-term weight + slug boost<br/>sort by (-score, slug)"]
    C -->|"true"| P["derive corpus/model/chunker<br/>provenance"]
    P --> V{"stored generation<br/>matches provenance?"}
    V -->|no| B["serialize and build one<br/>complete generation"]
    V -->|yes| L["cosine over validated<br/>sqlite-vec chunks"]
    B -->|commit| L
    B -->|"error / rollback"| W
    L -->|"EmbeddingUnavailableError"| W["logger.warning: falling back"]
    W --> K
    L -->|"ranked chunks"| D["dedupe chunks to runbooks"]
    K --> O["top-k runbooks"]
    D --> O

Diagram in words: Keyword retrieval remains the default. Semantic retrieval derives provenance, reuses only a matching complete generation, serializes a full rebuild when needed, and returns to the keyword scorer if embedding, rebuild, or read validation fails.

Own the failure modes before trusting the upgrade:

• An empty result set.
• A confidently-wrong nearest neighbor: cosine distance measures how closely two vectors point the same way, so it always returns something.
• The endpoint outage above.

How do I evaluate retrieval quality?

The dataset grades itself: every incident already names its runbook, so the ground truth needs no hand labeling. Hit-rate@k is the fraction of incidents whose runbook appears in the retriever's top k results.

evals/retrieval_eval.py builds a query from each incident's title and summary, scores both retrievers at k=1 and k=3, logs the comparison to MLflow, and prints a verdict. It builds semantic vectors in a fresh temporary state directory and logs the eight index-provenance fields as MLflow parameters, so a retained result identifies the exact generation it measured.

This is the only command on the page that needs a model running, so it has three prerequisites: cd agents/python && mise run config:check passing, the optional profile installed with cd agents/python && mise run install:eval, and the embedding model pulled with ollama pull nomic-embed-text.

cd agents/python
mise run eval:retrieval

The first CPU request may need to load the embedding weights from disk. AGENT_EMBEDDING_TIMEOUT_S bounds that cold start separately (120 seconds by default), so ordinary MCP tools retain their tighter deadline.

Historical checkpoint (2026-07-12, dataset commit ad4854b, Ollama 0.31.2, nomic-embed-text blob sha256:970aa74c0a90):

On those ten incident queries, semantic retrieval lost at k=1 and only matched at k=3. This snapshot supported the conservative default but could not qualify a candidate using a newer Ollama runtime. It remains historical evidence rather than a universal ranking or current-release result.

Release checkpoint (2026-07-31, dataset commit 0ff4a7e, Ollama 0.32.5, nomic-embed-text manifest sha256:0a109f422b47 and blob sha256:970aa74c0a90):

The current candidate reproduced the historical ranking in a fresh state directory. Its MLflow run recorded 36 chunks, 768 dimensions, chunker markdown-h2-h3-v1, and corpus digest ff1ce6b7000d. Semantic retrieval still does not beat the baseline, so keyword retrieval remains the shipped default.

# simplified
def hit_rate(retrieve, k: int) -> float:
    """Fraction of incidents whose runbook appears in the retriever's top-k."""
    pairs = cases()
    hits = sum(expected in retrieve(query, k) for query, expected in pairs)
    return hits / len(pairs)

The semantic side calls the local embeddings endpoint, so this evaluation stays outside the offline test gate. Set AGENT_SEMANTIC_RETRIEVAL=true only if a current, provenance-recorded run beats the keyword baseline for your corpus.

And remember what you now own: embedding model/version, chunking, index rebuilds, deletion, poisoning defenses, and re-running this evaluation when the corpus changes.

How do you defend against retrieval injection?

A retrieved runbook is data, not instruction. Six rules keep a poisoned document from steering the agent:

• Index only reviewed sources and preserve provenance.
• Treat retrieved instructions as lower priority than system/runtime policy.
• Never let a runbook expand the tool allowlist or self-approve an action.
• Return/cite the source slug so a human can inspect it.
• Bound result count and content size.
• Include malicious or contradictory documents in adversarial tests.

What is actually inside my agent's context window?

Four things share one window, and every one of them is sent again on every turn.

Everything above is about storing memory; this is the half about spending the window that holds it. The runtime assembles the whole request, and on the local path the window is the modest usable context of Qwen3-4B through Ollama. For the AgentOps Agent, one request is composed of:

flowchart TB
    subgraph W["One request — everything competes for one window"]
        direction TB
        I["Instruction<br/>fixed cost, every turn"]
        S["Tool schemas<br/>fixed cost, every turn"]
        H["Session history<br/>grows every turn"]
        R["Tool results<br/>grows, and persists into history"]
    end
    W --> M[Model]
    M -->|"exceeds serving window"| F["Request rejected or<br/>oldest tokens truncated"]
    F -.->|"the instruction sits at the front"| I

The tool schemas are easy to underestimate: in the conversational reference the agent registers a dozen tools (four reads, two knowledge, two guarded actions, two memory, two skill tools). Each docstring you write for the model is prompt text on every single turn. Tool results are the other heavyweight — search_runbooks caps even a larger model request at three whole runbooks, and once returned they live on in the session history.

The two fixed costs are paid on turn one and never go away; the two growing ones eventually push you over the edge. The pinned evaluation route rejects an oversized request. Other serving paths may truncate the oldest tokens instead, and your instruction is the oldest content. Either outcome is a failure: an explicit provider error, or an agent that has forgotten its own rules.

How do I measure tokens per component?

Per turn, you do not have to guess. The after-model callback in budget.py reads usage_metadata from each response and accumulates it in session state.

It emits the running totals as agentops.tokens.session.* span attributes, plus an agentops.tokens counter that Prometheus scrapes. Watching prompt_token_count grow turn over turn shows exactly how fast history accumulates. The same telemetry backs cost attribution in 7.3. Costs.

Per component, there is no shipped breakdown. Measure it yourself: change one thing at a time and compare the first turn's input-token count.

What do I drop first when I run out?

Three levers, in order of least behavioral risk first, all demonstrable on the local stack:

1. Retrieval width. search_runbooks(query, limit=3) returns whole documents and caps a larger model-controlled value at three; the model rarely needs three runbooks to cite one. Lowering limit is the direct lever. Note the semantic option does not help here: it ranks heading-bounded chunks internally but still returns each match's full runbook via read_runbook(slug). Switching retrievers does not shrink what enters the window; only limit does.
2. Tool schema weight. Terser docstrings and fewer registered tools shrink every turn. Note that routing reads through MCP does not make their schemas free — discovered declarations still enter the context; the saving comes only from what you stop exposing.
3. Session history. The course ships no automatic summarizer, but AGENT_MAX_HISTORY_MESSAGES provides an automatic deterministic trim (next section). The complementary deliberate lever needs no configuration: end the session, after saving durable findings with save_incident_note, and let recall_incident_context carry them into the fresh session. That is long-term memory used as explicit history compression. Skills already apply the same idea: list_skills shows only names and descriptions until load_skill pulls one body in.

Every cut trades against quality: a terser docstring can cost a correct tool choice, a smaller limit can drop the right runbook. The eval set in 4.4. Evaluations is the check — re-run the trajectory scores after each reduction and treat a drop as the real price of the tokens you saved. The eval:cost tripwire in that chapter is the matching guard for the opposite direction: it flags a change that quietly inflates tokens.

How do you bound the history automatically?

The levers above are deliberate — you end a session or tune a limit by hand. AGENT_MAX_HISTORY_MESSAGES adds an automatic one for a session that must stay live.

The app plugin calls it from before_model_callback, meaning ADK runs it just before each model call. It retains recent messages and complete tool exchanges, replacing the older span with a single synthetic note. It is off by default (unset sends the full history), deterministic, and model-free, so the offline test gate stays exact:

def compact_history(callback_context: CallbackContext, llm_request: LlmRequest) -> LlmResponse | None:
    """``before_model_callback``: keep only the most recent messages in the prompt.

    Returns ``None`` (never short-circuits the model call); it only rewrites
    ``llm_request.contents`` in place. Disabled unless ``AGENT_MAX_HISTORY_MESSAGES``
    is set, and a no-op until the history is longer than that budget.
    """
    del callback_context  # compaction depends only on the outgoing request
    keep = settings.max_history_messages
    if keep is None:
        return None
    contents = llm_request.contents
    if len(contents) <= keep:
        return None
    cut = len(contents) - keep
    # A tool batch is indivisible: retain its call and every response, including
    # when the latest message is still a tool result. Advancing the cut either
    # loses fresh evidence or leaves an orphan result that providers reject.
    # The message target is therefore soft at this protocol boundary.
    if _has_function_response(contents[cut]):
        while cut > 0:
            cut -= 1
            if any(part.function_call is not None for part in contents[cut].parts or ()):
                break
    if cut == 0:
        return None
    llm_request.contents[:] = [_marker(contents[:cut]), *contents[cut:]]
    return None

It sits between the token budget and PII redaction in the callback chain, so a refused call does no work and redaction only ever runs on the messages that survive. The rewrite is ephemeral — ADK rebuilds the request from the stored session events every turn, so nothing is deleted from the session and no marker ever accumulates.

Two properties keep it safe as a default. It never leaves behind an orphaned tool result — a function_response whose matching call was dropped, which the model cannot interpret. It also trims rather than summarizes, so no extra model call is involved.

Because a single user turn can expand into several messages (a model tool call, a tool response, a model answer), set the budget with headroom. Use token telemetry to check the actual reduction. The message target is not a token ceiling; a large tool exchange can still exceed the intended window.

Why can a long conversation fail or forget its start?

Because the model's advertised maximum and the serving window are two different numbers. ollama show qwen3:4b-instruct prints the architecture's context length. Ollama chooses the default serving window from available VRAM:

• Less than 24 GiB: 4K.
• At least 24 GiB but less than 48 GiB: 32K.
• At least 48 GiB: 256K.

OLLAMA_CONTEXT_LENGTH overrides that allocation. Run ollama ps while the model is loaded to see the window it actually received.

When the composed prompt exceeds that window, the current pinned OpenAI-compatible evaluation route returns HTTP 400. A serving path that truncates instead drops old content first. Since the instruction sits at the front, that path can shed the operating rules themselves, so grounding and approval discipline degrade exactly when the conversation is longest.

Detect it locally: the per-session input-token attributes keep climbing while the Ollama log reports a context error or truncation. Fix it explicitly with any of four levers:

• Raise OLLAMA_CONTEXT_LENGTH to what your hardware's memory allows.
• Shrink the context with the levers above.
• Bound the history with AGENT_MAX_HISTORY_MESSAGES so old turns are trimmed to a note instead of silently truncated.
• Set AGENT_MAX_TOKENS_PER_SESSION so a session ends with an actionable message before it degrades silently.

How would you add an eighth runbook and re-measure retrieval?

Optional exercise: grow the knowledge base, then check that retrieval still finds the right document.

• Mode: keep.
• Goal: add one runbook to the seven in agents/data/runbooks/, point an incident at it, and confirm the keyword scorer still returns the right runbook for every incident.
• Files to touch: a new agents/data/runbooks/<slug>.md with the same ## headings as the existing files, the incident's runbook value in agents/data/sql/seed.sql, and a new case in agents/python/tests/test_memory.py.
• Preflight: choose a new slug, require test ! -e agents/data/runbooks/<slug>.md, and require git diff --quiet -- agents/data/sql/seed.sql agents/data/incidents.db agents/python/tests/test_memory.py.
• Rebuild the seed: cd agents/data && mise run build, then mise run check — it fails if any incident names a runbook file that does not exist.
• Gate that proves completion: cd agents/python && uv run pytest tests/test_memory.py passes with a case asserting your slug is the top hit for a query phrased in its own words. With the embedding model pulled, mise run eval:retrieval is optional model-backed evidence rather than the deterministic gate.
• Final state: keep the new runbook, seed SQL, regenerated incidents.db, and test together; mise run check:data must pass and git status --short must contain only those intended files.

What proves this page worked?

cd agents/python
uv run pytest tests/test_memory.py tests/test_longterm.py tests/test_retrieval.py

Verify exact-slug lookup, deterministic tie ordering, no-match behavior, non-positive limit handling, path traversal rejection, per-user note isolation, redaction before persistence, provenance-triggered rebuilds, transactional rollback, concurrent first use, and the keyword fallback when embeddings are unavailable. Then manually compare the top result for database connection pool exhausted with the runbook source.

That focused subset exits cleanly. mise run test is the separate complete-suite gate that must clear 95% combined line-and-branch coverage.

You are done when:

• Every test in the three files above passes.
• get_runbook refuses a traversal slug: test_get_runbook_rejects_path_traversal passes, and the tool returns an error listing the available runbooks instead of file content.
• The top result for database connection pool exhausted matches, line for line, a file you can open under agents/data/runbooks/.
• You can say which of the six stores in the first table holds a given fact, which session-state prefix keeps a value for one invocation only, and what AGENT_MAX_HISTORY_MESSAGES changes about the prompt.

Continue to 3.5. Workflows when you can name, for any fact the agent needs, which of the six stores is supposed to hold it.